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Most generic performance tools display only system-level performance data us- 
ing 2-dimensional plots or diagrams and this limits the informational detail that 
can be displayed. Moreover, a modern relational database system, like Oracle, 
can concurrently serve thousands of client processes with different workload char- 
acteristics, so that generic performance-data displays inevitably hide important 
information. Drawing on our previous work, this paper demonstrates the applica- 
tion of Barry007 multidimensional visualization to the analysis of Oracle end-user, 
session-level, performance data, showing both collective trends and individual per- 
formance anomalies. 



INTRODUCTION 



Like most other graphical performance monitoring tools, 
the performance visualization (PerfViz) mindset in the 
Oracle database world is currently very two-dimensional. 
The typical performance dashboard (Fig. [T]) comprises 
standard strip charts with time running along the x-axis 
and the aggregation of certain database metrics posi- 
tioned on the y-axis. While this is often sufficient to get 
a rough idea of overall system performance, a dashboard 
view can be completely misleading [MM07]; especially 
when it comes to analyzing what individual Oracle pro- 
cesses are actually doing inside the database. 

The purpose of this paper to demonstrate how a multi- 
tude of Oracle performance metrics can be compressed 
into 2- and 3-dimensional visualizations by applying vari- 
ous barycentric coordinate transformations; collectively 
referred to as Barry007. In previous work |Gun92l 
JGO/I IGJ07I IGun08] , we have applied barycentric coor- 
dinates to visualizing multiprocessor utilization (Barry3 
coordinate system), web application response time data 
(Barry3 coordinates) and network-segment utilization 



t Copyright © 2008 Gunther, Poder. All Rights Reserved. This 
document may not be reproduced, in whole or in part, by any 
means, without the express permission of the authors. Permission 
has been granted to CMG, Inc. to publish in the Proceedings and 
the associated CD. Draft of September 15, 2008 




Figure 1: Oracle llg Enterprise Manager 



(Barry4 coordinate system). The numeric value N in 
the name BarryN indicates the number of performance 
metrics the can be visualized simultaneously. In the sub- 



sequent sections, we report on the latest application of 
Barry007 coordinates to the Oracle Wait Interface met- 
rics [Pod08], which are the primary performance indica- 
tors for Oracle database analysts. 



2 CURRENT ORACLE INTERFACE 

Oracle Enterprise Manager, shown in Fig. [T| is a per- 
formance monitoring tool commonly used by the Oracle 
database analyst (DBA). Its focus is on the database 
(DB) response time components and it facilitates easy 
data drill-down and navigation. However due to its 2- 
dimensional layout, it tends to obscure the visual identi- 
fication of performance anomalies or performance trends 
of individual Oracle sessions or session groups. This lim- 
itation can lead to a situation where 1000 application 
sessions with optimal performance can skew the system- 
wide statistics enough that the deleterious symptoms of 
100 suboptimal application sessions remain unnoticed. 

Oracle maintains its performance data as relational ta- 
bles in memory, known as v$ (pronounced "v-dollar") 
tables, which can be queried using SQL calls. Oracle has 
had session-level response-time instrumentation built in 
since early 1990s. It is called the Oracle Wait Interface 
(hereafter, OWI) and it presents a view of the wait timing 
information contained in the v$ tables. 

Oracle DB OS syscall Disk queueing 
execution execution and service 
(on CPU) (on CPU) (off CPU) 
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begin OWI wait 
T = gettimeofdayO 
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T 1= gettimeofdayO 
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Figure 2: Definition of OWI waiting times 
A word of caution may be appropriate here for those not 



familiar with Oracle OWI nomenclature. What is termed 
"wait time" in OWI is actually a response time [wJH03]. 
In Fig. [2j each OWI wait metric (R w ) is the sum of all 
actual wait times (Wi), in the sense of waiting for service, 
measured during i intervals and the corresponding service 
times (Si). More formally, we can write: 

i i 

There are some exceptions to this. For example, 
dbxpilpct is purely a measure of processor service time 
(execution cycles) without any Wi components. This is 
determined by what instrumentation and data sources 
are available to the OWI. See Appendix [A] for further 
discussion of this point. 

The response time data is gathered at database-session 
level, allowing detailed performance analysis of individ- 
ual database users. This presents many opportunities for 
performance diagnostics, such as deterministically diag- 
nosing performance problems for individual DB users who 
are experiencing high latency, regardless of the fact that 
the database appears to be running efficiently. Collecting 
detailed performance information in the OWI helps to re- 
move the mystery associated with performance problems 
by providing numerical evidence. 

However, due an extra dimension in the collected data, 
viz., the need to view users and sessions together, we run 
into a limitation for effective data presentation. Two- 
dimensional plots, like those in Fig. [T] quickly become 
cluttered when there is more than just a handful of ac- 
tive individual sessions. If each Oracle metric is assigned 
to an independent coordinate axis, then the greater the 
number of metrics in the performance instrumentation 
dataset, the greater the number of coordinate axes re- 
quired to view the result set. 

One answer to this problem is to provide the visualiza- 
tion tool with an interactive capability, that allows the 
analyst to navigate around in the data in real time, drill 
down, and slice-and-dice the data as desired. Essentially, 
one would just observe the data from multiple viewing 
angles. This is equivalent to creating new dimensions 
into the visualization by representing it multiple times 
over time. Even with that capability, such tools would 
still only be reactive, in the sense that the performance 
analyst or DBA would need to be aware of the problem 
first, before applying the tool from various viewing an- 
gles to "look into" and thereby diagnose the issue. A 
classic Catch-22 situation. Nonetheless, this approach is 
very similar to the first implemented in Tukey's PRIM- 
9 [Tuk88j tool, later modernized as MacSpin [AWD88] 
and now embedded in tools like Mathematica. 
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With these limitations as motivation, we now turn to the 
dimensional compression offered by barycentric coordi- 
nates. In the next section, we briefly review the concepts 
underlying the barycentric coordinate system. The inter- 
ested reader can find more background in Ref. [JG07]. 



3 BARYCENTRIC COORDINATES 



We introduce the concept of barycentric coordinates by 
showing that the locus of a point in the plane (i.e., 
two dimensions), defined by barycentric coordinates, is 
bounded by sides of an equilateral triangle, assumed to 
have height h = 1 for simplicity. 

Referring to Fig. [3] the location of any point inside 
the triangle can be determined by the lengths of the 
three arrows perpendicular to each side of the triangle. 
These arrows are the barycentric coordinates. Identify- 
ing each arrow-length by pi, p 2 and p 3 , the centroid in 



Fig. 3(a) corresponds to the height divided into three 
equal lengths: pi = p 2 = p 3 = 1/3- The centroid is the 
center-of-mass or balance point if equal weights were to 
be placed at each vertex. 

If instead, the weights were unequal, then the balance 
point would need to be shifted towards the heavier 
weights. As shown in Fig. |3(b)| the sum of arrow-lengths 
still equals the height of the triangle: 



P1+P2+ P3 



h = 1 



(2) 



Equation ^ is called a sum rule and it applies to any 
point interior of the triangle. It is an invariant because 
the three arrows partition the area of the triangle into 
three subareas that must, by definition, sum to the total 
triangle area. 

The barycentric coordinates for N performance metrics 
or N degrees of freedom is a d = N — 1 simplex. For 
example, N = 3 metrics can be represented in a 2- 
dimensional equilateral triangle because such a triangle 
is a d = (3 — 1) = 2-simplex. N = 4 metrics corresponds 
to a 3-simplex or tetrahedron (See Fig. [8j) . The virtue of 
the simplex-based coordinate system is that is contains 
the extra degree of freedom for free! This is where some 
of the compression, mentioned in Sect. [2] comes from. 

In general, a d-simplex has N = d + 1 faces. The cor- 
responding N barycentric coordinates are defined as the 
lines perpendicular to each face. Each vertex Vi in Fig. [3] 
has Cartesian coordinates (Vf x , Vi y ), where i = 1,2, 3. If 
we choose the Cartesian coordinates (l/v3, 1), (0,0) 
and (2/v3,0) for V\, V2, and V 3 , respectively, then the 




(a) The centroid or center-of-mass is the balance point of the 
triangle that would pertain if equal weights were placed at each 
vertex. It sits at a distance h/3 from each side. Hence, the 
barycentric coordinates are pi = P2 = P3 = 0.3333 when h = 1 




(b) If the vertex weights are different, the point p must be 
relocated to compensate for this imbalance. Here the vertices 
Vi and V2 are "heavier" than V3, so the barycentric coordinates 
become pi = 0.62, p 2 = 0.28, p 3 = 0.10 with h = 1. The 
dashed lines intersect at the centroid 



Figure 3: Barry3 coordinates for a 2-simplex. 
Cartesian coordinates of the point (p x ,p y ) are given by: 

(Px,Py) = (PlV lx + p 2 V 2x + p 3 V 3x , 

PlVly + P2V 2 y + P 3 V 3 y) . 

Consider the centroid in Fig. 3(a)| Its barycentric coor 



(3) 
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dinates are p\ = p 2 = p 3 
Cartesian coordinates are: 



1/3, so its corresponding 



(PlVlx + P2V2X + P3V3X, PlVly + p 2 V 2 y + P3V 3 y) 



+ + 



(Px,Py) 



11 12 1 

,37! + + 37!' 3 

1 1 
^ 3 

This result can be checked in Fig. 3(a) The vertex 
V\ is located at l/v3 = 0.5774 on the z-axis, so the 
Cartesian coordinates of the centroid are (p x ,p y ) = 
(0.58, 0.33). Similarly, in Fig. [3(b)] Pl = 0.62, p 2 = 0.10 
and ps = 0.28, so the three arrows meet at (p x ,p y ) — 
(0.47,0.62). 



4 OWI METRICS TEST CASE 



We now turn to the representation of OWI wait-class 
metrics in Barry3 coordinates. As proof-of-concept, we 
consider first aggregating several OWI metrics into just 
the three metrics needed to define the Barry3 axes: pi, 
p 2 and p 3 . The following eight OWI Oracle wait classes: 



USERIO.PCT 

SYSTEMICLPCT 

APPLICATION.PCT 

COMMIT.PCT 

CONCURRENCY_PCT 

CONFIGURATION.PCT 

NETWORK_PCT 

OTHER.PCT 



along with db_cpu_pct, were aggregated into 3 composite 
classes: 



1. WAIT_PCT = APPLICATION_PCT + COMMIT_PCT + CONCUR- 
RENCY.PCT + CONFIGURATION.PCT + NETWORK.PCT + 
OTHER.PCT 

2. IO-PCT = USERIO-PCT + SYSTEMIO_PCT 

3. DB_CPU_PCT 



WAIT_PCT 




(a) Stacked-chart of the three composite OWI-test metrics plotted 
as a function of snapshots identified by a SNAPJD 




00:00 02:00 04:00 06:00 

(b) OWI test metrics in Fig. |4(a)"| displayed as a time series 

Figure 4: Composite OWI metrics for test case 



Fig. |4(b)| shows the same three OWI combined metrics 
plotted as a time series. We observe immediately that 
the io_pct metric remains relatively constant at around 
10% over the entire measurement period (T ~ 7 hours). 
The db_cpu_pct metric is falling slowly, while wait_pct 
increases proportionately. The latter metrics cross one 
another at about 4.5 hours into the measurement win- 
dow. 



shown as a stacked chart in Fig. 4(a) All of these metrics 
are drawn from the v$system_wait_class table and 
have been normalized to percentages (pct). 



Fig. 4(a) verifies graphically that the three composite 



OWI-test metrics, db_cpu_pct, io_pct and wait_pct, do 
obey the Barry3 sum rule 

A performance analyst, however, is more likely to want 
to see these data clearly separated into their respec 
tive components, rather being stacked as in Fig. |4(a) 



5 BARRY3 OWI REPRESENTATION 



We are now in a position to transform the OWI test 
metrics of Fig. 4(a)| into a Barry3 representation. The 
Barry3 axes associated with Fig. [3] are: 



1. dbxpilpct: "north-pointing" red arrow 

2. wait_pct: green arrow pointing "south-east" 

3. IO-PCT: blue arrow pointing "south-west" 
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The corresponding Barry3 representation is shown in 
Fig. [5] This particular choice of axes is entirely arbi- 
trary and any actual choice should be at the disposal of 
the user. Each snapjd sample in Fig. 4(a)| corresponds 
to the location of the dot in Barry3. Fig. 5(a) shows 
such a point with its barycentric coordinates {pi,Pi,Ps) 
corresponding to snapjd = 8. 

DB CPU PCT 




10 PCT 



WAIT PCT 



(a) SNAPJD sample number 8 of the composite OWI-metrics in 
Fig. |4(a)| represented by a dot inside the Barry3 triangle 

DB_CPU_PCT 

A 




10 PCT 



WAIT PCT 



(b) Trilinear coordinates are a visual aid for assessing the contribution 
of each OWI-metric to the dot position inside Barry3 



Figure 5: Single snapshot of composite OWI-metrics 
from Fig. |4(a) displayed as a dot in a Barry3 representa- 
tion. A succession of such snapshots produces animation 



have Oracle processes executing than it is to have them 
waiting on (non-computational) resources. But this is 
matter of personal preference when it comes to visual- 
ization and beyond the prototype display shown here, it 
should be a user-definable option in any released PerfViz 
tool. 



In Fig. |5(b)| the barycentric coordinates in Fig. 5(a) have 
been replaced by trilinear coordinates [Har99]. Such a 
static coordinate system might be preferred for qualita- 
tively assessing the contribution of each OWI-metric to 
the dot position. 

A succession of OWI snapshots from Fig. [5] produces 
animation in Barry3. See http://www.perf dynamics. 



com/Test/owiCompB3.gif ). The dot moves inside the 
triangle. Each time-step is displayed as an odometer 
value in the lower part of the triangle. As we soon see 
in an animation, the dot stays close to the edge of the 
Barry3 triangle. This corresponds to the constant 10% 
icxpct time seen in Fig. |4(b)[ Gradually, the dot drifts 
down, along the edge, as the percentage of wait_pct time 
increases at the expense of dbxpilpct time. 



6 MULTIPLE OWI SESSIONS 



Next, we extend the proof-of-concept described in 
Sects. [4] and [5] to render multiple Oracle sessions in 
Barry3. For the purpose of Oracle data collection and 
cross-checking, we used the following tools: 

Sesspack: Sesspack is a session-level data collection 
tool for Oracle. It samples relevant v$ tables pe- 
riodically and stores the output to persistent tables 
for future analysis. The benefit of Sesspack relative 
to the usual Oracle data collectors lies in the level of 
detail captured and its flexibility to allow the ana- 
lyst to specify both the sessions to monitor and the 
metrics to measure. 

PerfSheet: PerfSheet is an Excel-based PerfViz tool 
which facilitates a convenient visualization of the 
result set of any SQL query, including those made 
against Sesspack data. The relevant data is auto- 
matically fetched to PerfSheet, after which a user 
is free to view that data in Excel pivot charts as 
desired. 



We have chosen to make dbxpilpct the apex of the 
Barry3 triangle because it is presumably more desirable to 



See Ref. [Pod08j for more details. In the current pro- 
totypes presented here, the primary source of OWI data 
for Barry007 is Sesspack. The Barry3 axes selected from 
the available OWI metrics are: 
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1. Cpu.usage: Oracle session is executing on a CPU. 

2. idle: Session is idle waiting for next client request. 

3. db.wait: Other waits for I/O, DB locks, etc. 

Note, that although these are not reported as percent- 
ages (as in Sect. [5]), all three times must sum to the 
sample interval of 1000 ms, or 1 second, for each session 
such that: 

CPU-USAGE + IDLE + DB-WAIT = 1000 ms (4) 

in agreement with ([2]). An example data set with 60 
simultaneous OWI sessions, displayed in Barry3 coordi- 
nates, is shown in Fig. [6] 



CPU.USAGE 




IDLE DB_WAIT 

Figure 6: A snapshot of 60 OWI sessions displayed si- 
multaneously as colored dots in Barry3. Note the slightly 
different choice of OWI metrics from those in Fig. [5] 

There is an important difference between the datasets 
used here and those in Sect. [4] The test-case OWI data 
do not contain an idle component. Therefore, we can 
only visualize the components of time that the DB spends 
servicing a user call viz., the "DB Time," in Oracle par- 
lance. It is not possible to directly map this back to 
wall-clock time, because we do not know how much idle 
time was incurred. 

In the session-level data being discussed here, the idle 
time is measured, so we can directly map the measured 
times to wall-clock time and any "missing" or multiply- 
accounted time indicates that some measurement error 
was involved. We shall return to the issue of measure- 
ment errors in Sect. HI 

In Fig. [6] (one frame of an animated time series) we see 
immediately that 4 sessions are completely idle (lower- 



left vertex), 2 sessions are 50% idle (dashed-diagonal blue 
line second from left), a cluster of sessions that are about 
10% CPU busy (group of colored dots at bottom center- 
right), and 2 sessions are waiting on the DB for 100% of 
the time (lower-right vertex). 




Figure 7: Stacked view of Fig. 6 



Note that we just counted 2 points at the db_wait_pct 
vertex in Fig. [6] whereas only a single colored dot can 
be seen. This visual anomaly raises an important point; 
one that would have to be addressed in any real Barry3 
visualizations. In this prototype display, one dot overlaps 
the other. We can verify this effect with the stacked view 
in Fig. [7] The db.wait metric (shown in mauve) has two 
spikes that reach 100%. In an animated Barry3 display, 
overlapping points are less of an issue because the points 
are in constant motion. In a static Barry3 view, although 
overlapping is more significant, it could be corrected by 
adding a small offset to the positions of any overlapping 
dots. 



7 BARRY4 OWI REPRESENTATION 

We can further resolve the DBwait time in waiting for 
DB locks and latches and waiting on other resources e.g., 
physical I/O. The 4 OWI metrics then become: 

1. CPU-Usage: Oracle session is active and executing on 
a CPU. 

2. dbxontention: Session is active but waiting on 
database locks or latches. 

3. DB-WAiT: Session is active, but waiting on other waits 
like disk and network I/O. 

4. idle: Session is inactive and waiting for next client 
request. 



G 



so we need to consider a Barry4 representation. Here, 
active means that the Oracle session is serving a user 
request. 

CPU_USAGE 




DB_CONTENTION 

Figure 8: Snapshot of 60 OWI sessions in Barry4. Two 
clusters of points stand out at this viewing angle 

Although these OWI metrics are not reported as percent- 
ages, they must sum to the sample interval of 1000 ms 
or 1 second in order to meet the sum rule condition ([2]). 

CPUJJSAGE + DB.CONTENTION + DB.WAIT + IDLE = 1 S (5) 

Barry4 is a 3-simplex or tetrahedron shown in Fig. [8] We 
see that 2 sessions are essentially inactive (near 100% 
idle). There are two clusters of points that are very no- 
ticeable. One of the clusters is experiencing a moderate 
degree of dbxontention, while the other cluster is expe- 
riencing a moderately high degree of db_wait time. In a 
presentation we would be able to demonstrate how the 
tetrahedron in Fig. |8]can be swiveled with the mouse to 
facilitate views from different perspectives. 



8 OWI INSTRUMENTATION ISSUES 

In order to build a multidimensional performance visual- 
ization solution which can make sense out of the large 
amount of instrumentation data, we need to focus on 
the end user response time. Simply monitoring the total 
response time, however, is not sufficient for diagnosing 
causes of performance problems, thus we need response 



time breakdown to individual events like CPU time, 10 
waits, DB lock waits, etc. 

We also need full response time accounting, such that 
no component of the user response-time is missing from 
the total. We also need DB session level instrumentation 
with the ability to take snapshots of individual sessions 
and Sesspack [Pod08] provides that data for the Barry007 
representation. 

Since Barry007 imposes the constraint of a sum rule ([2]), 
it also exposed previously unseen limitations in Oracle's 
performance instrumentation viz., the OWI metrics in 
Q and ([5]) do not always consistently add to 1000 ms. 
There are occasional bugs with OWI response time com- 
ponents unaccounted for. In addition, CPU preemption 
and run-queue waiting time in a multiprocessor or mul- 
ticore are not accounted by default, across all platforms. 
So here is an unexpected diagnostic benefit of PerfViz. 
Without the aid of good visualization tools such defects 
can easily go unnoticed and continue to act as an un- 
known error source in any performance analysis and ca- 
pacity planning. 

The latest version of OWI reports 959 different wait 
events, which Oracle has grouped into a smaller num- 
ber of wait classes: 



SQL> select wait_class, count (*) 

2 from v$event_name 

3 group by wait_class; 



WAIT_CLASS C0UNTO) 


Concurrency 


26 


System 1/0 


23 


User 1/0 


22 


Administrative 


51 


Other 


630 


Configuration 


21 


Scheduler 


3 


Cluster 


47 


Application 


15 


Queue ing 


4 


Idle 


80 


Network 


35 


Commit 


2 



We would simply need to further collapse these wait 
classes in order to provide the appropriate input for 
Barry007 analysis. 



9 CONCLUSION 

In this paper we have presented another application of 
performance visualization based on the dimensional com- 
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pression offered by barycentric coordinates. The sum- 
rule ([2]) imposes a very severe constraint on the perfor- 
mance metrics that can be displayed in the Barry007 style 
and this makes it challenging to find appropriate appli- 
cations. As we have shown here, we can now add Oracle 
Wait Interface (OWI) to a growing list of metrics that 
are amenable to both Barry3 and Barry4 display. 

Inaccuracies in collected performance data is an often 
overlooked subject because there is a widespread ten- 
dency to believe measured numbers and unless one has 
the right tools and techniques, the process of determin- 
ing the level of measurement error can be a very difficult 
one. Another benefit of our approach was the ability to 
visually identify missing times in OWI. Without the aid 
of good PerfViz tools such defects can remain unnoticed 
and continue to introduce errors into any database per- 
formance analysis. With tools like Barry007 that enable 
data discovery rather than just data reporting, measure- 
ment error is more easily revealed and quantified. 

By comparison with standard 2-dimensional tools that 
simply display time on the x-axis, Barry007 can apply an- 
imation to represent the time-development of each per- 
formance metric, thus freeing up the x-axis to display a 
different metric. Using current performance tools, the 
DBA is often forced to address each of the following 
performance issues separately: 

1. Display the current performance state of a database 
with session-level detail 

2. Early detection of performance anomalies 

3. Detecting application-level performance trends 

Barry007 could have a positive impact in Oracle perfor- 
mance management since it provides a solution for all 
three of these issues. 

More generally, in the arena of PerfViz, a stalemate exists 
between performance-tool vendors who are disinclined 
to invest in engineering development when they do not 
see any demand, and performance analysts who are not 
aware of what they are missing in terms of better PerfViz 
for solving performance management problems. 

All the illustrations and demonstrations in this paper were 
created with Mathematica 6.0 and are therefore only pro- 
totypes. Naturally, prototypes suffer from the kinds of 
limitations discussed in Sects. [6] and [7] Nonetheless, our 
hope is that these prototypes, as well as others [JG07], 
will provide further incentive to improve the kind of ex- 
ploratory PerfViz tools available to both Oracle DBAs in 
particular and performance analysts in general. 
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A APPENDIX 

Here, we discuss in more detail Oracle's approach to de- 
composing application (end user) response time into its 
fine-grain components. Some examples where this is 
done, include: Oracle server process waiting for 10 to 
be completed, Oracle waiting for a lock to be released 
and time spent executing on a CPU. 



A.l OWI Wait Time 

The wait times are measured inside Oracle. Whenever 
Oracle is about to issue a system call which could block 
and cause the process go off CPU, Oracle gets a system 
timestamp using gettimeofday() just before issuing the 
system call. Another timestamp is taken when a system 
call has completed and Oracle process has got back onto 
CPU. This has the following implications: 

• Some of the Oracle "wait" time includes time spent 
executing on CPU as issuing the system call requires 
some CPU cycles. Also, the CPU busy time spent by 
OS serving the system call (in kernel mode) is also 
attributed to the Oracle process. Also, sometimes 
(like readO system calls from OS filesystem cache) 
do not cause the process go off CPU at all as the 
read can be satisfied from server main memory. In 
such cases Oracle reports short waits, while all this 
time was actually spent on CPU. 

• Another implication is that if a process has been 
scheduled off CPU due blocking system call, then 
it needs to get back onto CPU to take the wait 
end timestamp. If there is some scheduling latency, 
it will be implicitly attributed to the wait event as 
well, since Oracle doesn't know about scheduling 
latency on most platforms. 



A. 2 OWI CPU Time 

On a Unix system, an Oracle server process executes 
getrusageO system calls at the end of every database 
call (or every 5 seconds for long DB calls) for getting CPU 
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usage numbers for that process from OS. CPU schedul- 
ing latency is not included in Oracle's CPU time statistics 
as most OS-es don't account this. As CPU and Wait 
times are gotten using different methods, from different 
sources, this leaves some room for double counting er- 
rors. 



A. 3 Barycentric Sum Rule 

In the total (wall-clock) elapsed time, we have three com- 
ponents: 

1. Oracle measured waits: the difference between Or- 
acle's wait begin and wait end timestamps. 

2. OS measured process CPU time usage. Oracle gets 
this data from OS. It can be interpreted as the min- 
imum elapsed time Oracle spent in CPU service. 

3. Unaccounted time (T u ). This includes all other 
waits: 

T u = R - Wo - B 

where R is the end-to-end user response time, Wo is 
the wait-time as measured by the Oracle RDBMS, 
and B is CPU busy-time as measuredby the OS. 
T u includes scheduling latency, measurement errors, 
instrumentation bugs. 
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